Skip to content

feat(llm): record and aggregate model usage across consumers - #320

Merged
enkerewpo merged 14 commits into
syswonder:devfrom
agicy:feat/llm-usage-metering
Nov 15, 2026
Merged

enkerewpo merged 14 commits into
syswonder:devfrom
agicy:feat/llm-usage-metering

Conversation

@agicy

@agicy agicy commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Problem

Robonix's model consumers call OpenAI-compatible endpoints directly, and each one keeps its own logging. There is no shared record of which consumer used which model, how many tokens were billed, how long a call took, or how it failed. Pilot has internal usage logs, but Scene, memory, memsearch and the VLM verifier do not expose usage, so an operator cannot attribute model spend or compare outcomes across components.

User-visible impact

rbnx boot creates <deploy>/rbnx-boot/llm-metrics/ and exports ROBONIX_LLM_METRICS_DIR to every child. Each model call made by Pilot, Scene, memory, memsearch and the VLM verifier appends one JSON line with consumer, operation, endpoint host, model, outcome, attempts, token counts and latency. rbnx llm-usage sums those records by consumer, operation, host and model and reports calls, attempts, outcomes, tokens and p50/p95/max latency; --json prints one JSON object per group. Records persist across boots and are removed only with rbnx clean -f --llm-metrics. Standalone consumers that do not set the directory write nothing. The change adds no proxy, service, port or model-catalog dependency.

Implementation

pylib/robonix-api/robonix_api/llm/metrics.py adds Call, usage_from_response() and the JSONL writer. Call records success, HTTP errors, timeouts, cancellation and unreadable 2xx bodies; a metrics write failure is logged once and never fails the model call.

system/llm-metrics is the Rust counterpart. Call writes on drop, Usage normalizes OpenAI and DeepSeek cache fields, and Status records the real outcome. Both writers use robonix_api/llm/contract/usage_cases.json.

Pilot records plan and summary calls, including stream completion, abandonment, timeout and truncated replies. Scene records relations, image_relations, caption and perception; memory records search, observe and answer; the verifier records judge.

memsearch now sends its own chat-completions request from COMPACT_PROMPT so the compact call has token counts. The previous MemSearch.compact call returned no usage. The memsearch dependency range is pinned and httpx is added.

rbnx creates the metrics directory, exports it to children, adds rbnx llm-usage, and gives rbnx clean -f a separate --llm-metrics switch. Scene mounts the directory writable. CI adds a robonix-api unit-test job; the existing workspace job covers Rust.

Validation

  • PYTHONPATH=pylib/robonix-api python3 -m unittest pylib/robonix-api/tests/test_llm_metrics.py: 10 passed.
  • PYTHONPATH=pylib/robonix-api:services/memory python3 -m pytest -q services/memory/tests/test_llm_search.py services/memory/tests/test_observe.py: 20 passed.
  • PYTHONPATH=pylib/robonix-api:services/verifiers/vlm_verifier:services/verifiers/vlm_verifier/tests python3 -m unittest test_verifier.ParsingTests test_verifier.RequestTests: 22 passed.
  • PYTHONPATH=pylib/robonix-api:services/memsearch python3 -m unittest discover -s services/memsearch/tests -p 'test_lifecycle.py': 7 passed.
  • cargo test --workspace --all-targets: passed.
  • cargo fmt --all: clean.
  • cargo clippy --workspace --all-targets -- -D warnings: clean.
  • python3 scripts/check_commit_authorship.py --base origin/dev --head HEAD: 6 commits passed.

No Webots run. This change only observes model calls, except memsearch now issues the same chat-completions request itself; its lifecycle tests cover that path.

Configuration and migration

ROBONIX_LLM_METRICS_DIR enables recording. rbnx boot sets it by default; an existing value wins. RBNX_BOOT_ID is copied into each record when present. Setting the directory to an empty value disables recording. rbnx clean -f keeps the directory unless --llm-metrics is passed. No capability contracts, generated code, wire protocols or model requests change, except the memsearch request described above. When the model-catalog launcher lands, it must pass ROBONIX_LLM_METRICS_DIR and RBNX_BOOT_ID through its environment filter.

Related issues

None.

CLI output

Fixture run of rbnx llm-usage -d <fixture>:

consumer       operation        host                     model                     calls attempts      input     output     cached  reasoning unreported   lat_n   p50_ms   p95_ms   max_ms  status
pilot          plan             h                        m                             2        3         10          3          0          0          1       2      100      300      300  ok=1 timeout=1

rbnx llm-usage --json emits the same group with latency_samples, latency_p50_ms, latency_p95_ms and ``latency_max_ms`.

Review notes

The memsearch change replaces a library call with an in-tree request to obtain usage; reviewers should check prompt and indexing parity against the pinned memsearch version. The Pilot change is larger because a stream needs a real completion/abandonment/timeout outcome for the record; the new tests cover those paths.

@github-actions github-actions Bot added type:feature New feature (feat:) comp:pilot system/pilot comp:scene system/scene comp:memory services/memsearch comp:rbnx tools/rbnx comp:api pylib/robonix-api comp:docs docs/ and READMEs comp:ci .github/ workflows labels Nov 8, 2026

@enkerewpo enkerewpo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested 7e407c5 locally. On the PR head: cargo clippy --workspace --tests -D warnings is clean and cargo test --workspace --all-targets passes (420 tests, including chat_stream_records_each_outcome and the robonix-llm-metrics contract tests); robonix-api metrics 10 OK, memory 20 passed, vlm_verifier 22 OK, memsearch lifecycle 7 OK. Scene's suite gives the same result as on dev (408 passed; the failures and collection errors are missing local modules, identical on both).

I also checked _compact against memsearch 0.4.11: same prompt, chunk joining, memory/YYYY-MM-DD.md path, heading and re-indexing, and MemSearch.store is public there. It also fixes the old call, which passed model=/base_url=/api_key= while 0.4.11's MemSearch.compact takes llm_model=/llm_base_url=/llm_api_key=.

One blocker: merged onto current dev (39ab370), robonix-pilot no longer compiles, see the inline comment. Please rebase onto dev and fix it; the rest looks good to me.

Comment thread system/pilot/src/vlm.rs
model: "local-test".into(),
configured_context_window_tokens: None,
prompt_cache_key: "local-test".into(),
metrics_dir: Some(metrics_dir.to_path_buf()),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After #321 VlmClient has a reasoning_effort field, so merged with current dev this initializer fails: error[E0063]: missing field reasoning_effort in initializer of vlm::VlmClient (cargo clippy --workspace --tests). Please rebase and add reasoning_effort: None, here.

@agicy
agicy force-pushed the feat/llm-usage-metering branch from 7e407c5 to ff25ada Compare November 15, 2026 13:05
@agicy

agicy commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

@enkerewpo

Thanks — rebased onto dev and fixed the blocker (reasoning_effort: None).

While rebasing I found the launcher env allowlist dropped ROBONIX_LLM_METRICS_DIR,
which silently disabled usage records under a catalog; fixed with tests. The new
robonix-api CI job also failed on the atlas binding test — it now skips when the
bindings or binary are absent, while the atlas job still runs it.

CI is green; fmt/clippy clean and the Python suites pass.

@enkerewpo enkerewpo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick rebase and the extra fix. The reasoning_effort blocker from my last review is resolved, and the ROBONIX_LLM_METRICS_DIR forwarding fix is a good catch: without it, every consumer started through the catalog launcher would silently drop its usage records.

Tested 3cdd09a (the current head, rebased on dev 4368021) locally:

  • Rust: cargo fmt --check and cargo clippy --workspace --tests -D warnings are clean; cargo test --workspace --all-targets passes 500 tests with 0 failures, including llm_usage, usage_matches_the_shared_cases and chat_stream_records_each_outcome.
  • Python: robonix-api 75 passed / 1 skipped, catalog contract 39 OK, Memsearch lifecycle 16 OK, Memory + Scene 51 passed, verifier 30 OK.
  • End to end: wrote records through robonix_api.llm.metrics.Call for success, HTTP 429, read timeout, a retried call and a 2xx HTML body, then summed them with rbnx llm-usage. Outcomes, attempts, tokens, provider_model and the --boot filter all come out as expected.
Rust: fmt, clippy, tests
$ cargo fmt --all -- --check
(no output, exit 0)
$ cargo clippy --workspace --tests -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 38.31s
$ cargo test --workspace --all-targets
test cmd::llm_usage::tests::sums_by_group_and_boot_and_skips_partial_lines ... ok
test usage_matches_the_shared_cases ... ok
test a_call_writes_one_record_with_the_last_reported_usage ... ok
test vlm::tests::chat_stream_records_each_outcome ... ok
...
total over all test binaries: 500 passed; 0 failed
Python suites (same commands as CI)
$ python -m pytest -q pylib/robonix-api/tests
75 passed, 1 skipped, 65 subtests passed in 0.34s
$ python -m unittest discover -s pylib/robonix-api/tests -p 'test_llm_*.py'
Ran 39 tests in 0.285s
OK
$ python -m unittest discover -s services/memsearch/tests -p 'test_lifecycle.py'
Ran 16 tests in 0.026s
OK
$ python -m pytest -q services/memory/tests/{test_llm_search,test_observe,test_service}.py system/scene/tests/{test_llm_catalog,test_runtime_protobuf_contract}.py
51 passed, 5 subtests passed in 0.56s
$ python -m unittest test_verifier.ParsingTests test_verifier.RequestTests
Ran 30 tests in 0.032s
OK
End to end: records written by robonix-api, summed by rbnx llm-usage

Records: 2× pilot chat 200 with usage, 1× pilot chat 429, 1× scene describe httpx.ReadTimeout, 1× memory embed with one retry, 1× memory embed returning a 2xx HTML body.

$ ls $ROBONIX_LLM_METRICS_DIR
memory-<pid>.jsonl
pilot-<pid>.jsonl
scene-<pid>.jsonl
$ head -n1 pilot-<pid>.jsonl
{"v":1,"consumer":"pilot","operation":"chat","host":"api.example.com","model":"demo-vlm","stream":false,"attempts":1,"status":"ok","http_status":200,"input_tokens":120,"output_tokens":30,"cached_input_tokens":null,"reasoning_tokens":null,"provider_model":"demo-vlm-0612","latency_ms":0,"ts_ms":<ts>,"boot_id":"boot-demo","pid":<pid>}
$ rbnx llm-usage
consumer       operation        host                     model                     calls attempts      input     output     cached  reasoning unreported   lat_n   p50_ms   p95_ms   max_ms  status
memory         embed            localhost:8000           demo-embed                    2        3         40          0          0          0          2       2        0        0        0  ok=1 transport_error=1
pilot          chat             api.example.com          demo-vlm                      3        3        240         60          0          0          1       3        0        0        0  http_error=1 ok=2
scene          describe         api.example.com          demo-vlm                      1        1          0          0          0          0          1       1        0        0        0  timeout=1
$ rbnx llm-usage --boot other
consumer       operation        host                     model                     calls attempts      input     output     cached  reasoning unreported   lat_n   p50_ms   p95_ms   max_ms  status

Non-blocking, for a later change if you like: rbnx llm-usage --json | head -n1 prints the first line and then panics with failed printing to stdout: Broken pipe. It is cosmetic and only happens when the reader closes the pipe early.

LGTM.

@enkerewpo
enkerewpo merged commit 1145b35 into syswonder:dev Nov 15, 2026
17 checks passed
@agicy
agicy deleted the feat/llm-usage-metering branch November 15, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:api pylib/robonix-api comp:ci .github/ workflows comp:docs docs/ and READMEs comp:memory services/memsearch comp:pilot system/pilot comp:rbnx tools/rbnx comp:scene system/scene type:feature New feature (feat:)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants